Skip to content

Perf: Optimize pages loading (Filecache path like approach) - #2549

Merged
max-nextcloud merged 3 commits into
mainfrom
feature/optimize-pages-loading-v2
Jun 30, 2026
Merged

Perf: Optimize pages loading (Filecache path like approach)#2549
max-nextcloud merged 3 commits into
mainfrom
feature/optimize-pages-loading-v2

Conversation

@Koc

@Koc Koc commented May 31, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

📝 Summary

This is alternative approach to #2390 that fixes same performance issue (closes #2380).

Benefits comparing to previous implementation:

  • no extra columns
  • no migration to re-process already existent page
  • no listeners
  • much simpler implementation

So, we're just load all necessary pages via simple query SELECT * FROM filecache WHERE storage_id = <storageId> AND path LIKE 'appdata_<instanceId>/collectives/<collectiveId>/%'

🖼️ Screenshots

image

Collective with 390 pages with various nesting level

🏚️ Before 🏡 After
6.31s 5.20s
image image
747 queries - depends on pages count and nesting level 50 queries - more or less constant
image image

🚧 TODO

🏁 Checklist

  • Code is properly formatted (npm run lint / npm run stylelint / composer run cs:check)
  • Sign-off message is added to all commits
  • Tests (unit, integration and/or end-to-end) passing and the changes are covered with tests
  • Documentation (README or documentation) has been updated or is not required

🤖 AI (if applicable)

  • The content of this PR was partly or fully generated using AI tools
  • The AI-generated content was reviewed, comprehended and tested by a human

@Koc Koc mentioned this pull request May 31, 2026
Closed
@Koc
Koc force-pushed the feature/optimize-pages-loading-v2 branch 4 times, most recently from 5d10baa to 5e6b2e9 Compare May 31, 2026 13:31
mejo-
mejo- previously requested changes Jun 9, 2026
View reviewed changes

@mejo- mejo- left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot @Koc, this looks really promising 🤩

I have some comments, but I'm genuinely curious what you think about the comments.

If you do further changes to the PR, could you do them in separate fixup commits (and don't force-push changes to the existing commit for now) so it's easier to review your changes?

Comment thread lib/Service/PageService.php Outdated
Comment thread lib/Model/FileInfo.php Outdated
Comment thread lib/Mount/CollectiveFolderManager.php
Comment thread lib/Mount/CollectiveFolderManager.php
Comment thread lib/Service/PageService.php Outdated
Comment thread lib/Service/PageService.php Outdated
Comment thread lib/Mount/CollectiveFolderManager.php Outdated
Comment thread lib/Service/PageService.php Outdated
Comment thread lib/Model/PageInfo.php Outdated
Comment thread lib/Model/PageInfo.php
@Koc Koc self-assigned this Jun 9, 2026
@Koc
Koc force-pushed the feature/optimize-pages-loading-v2 branch 6 times, most recently from 3c03b17 to 980fb42 Compare June 14, 2026 22:19
@Koc

Koc commented Jun 14, 2026

Copy link
Copy Markdown
Contributor Author

We have green pipeline 🎉

@Koc
Koc force-pushed the feature/optimize-pages-loading-v2 branch from 0ac5ab9 to 6937cb2 Compare June 15, 2026 09:37
@github-project-automation github-project-automation Bot moved this to 🧭 Planning evaluation (don't pick) in 📝 Productivity team Jun 15, 2026
@max-nextcloud max-nextcloud moved this from 🧭 Planning evaluation (don't pick) to 👀 In review in 📝 Productivity team Jun 15, 2026
@max-nextcloud max-nextcloud self-assigned this Jun 15, 2026
max-nextcloud
max-nextcloud reviewed Jun 15, 2026
View reviewed changes

@max-nextcloud max-nextcloud left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Proposing a refactoring of the getPagesFromFolder / buildPageInfoTree functions.

I'm pretty sure your approach works as well. I think this refactoring would still clarify things a lot. Would you be up for it or shall I give it a try and open a PR against this one?

Comment thread lib/Service/PageService.php Outdated
max-nextcloud
max-nextcloud reviewed Jun 15, 2026
View reviewed changes
Comment thread lib/Service/PageService.php Outdated
@max-nextcloud

max-nextcloud commented Jun 18, 2026

Copy link
Copy Markdown
Collaborator

Just tested this locally and performance wise it's really great. Managed to handle a collective with 2k pages well. Became somewhat slow from 3k on.
In comparison - on the same machine 700 pages were getting problematic before. So this seems to provide for approx 4-5 times as many pages.

@Koc

Koc commented Jun 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

@max-nextcloud @mejo- WDYT about adding a composite index to filecache table to columns storage_id, path? This can improve performance even more or large instances with millions files

@max-nextcloud

max-nextcloud commented Jun 20, 2026

Copy link
Copy Markdown
Collaborator

@Koc there's the fs_storage_path_prefix cache (except for postgres dbs):
https://github.com/nextcloud/server/blob/master/core/Migrations/Version13000Date20170718121200.php#L244-L246

Might be interesting to confirm it's being used though. On my local instance requests for > 1k files were way below 1 second. That seems good enough.

@max-nextcloud max-nextcloud mentioned this pull request Jun 20, 2026
Open
@max-nextcloud

max-nextcloud commented Jun 20, 2026

Copy link
Copy Markdown
Collaborator

From my point of view the next step would be to fetch the pages for even larger collectives in multiple requests.
I sketched my ideas about that here: #2581

@Koc

Koc commented Jun 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

@max-nextcloud heh we're using Postgres 😕 Why it's excluded?

I will try to work on your other comments next week

@Koc Koc mentioned this pull request Jun 21, 2026
Closed
4 tasks
@max-nextcloud

max-nextcloud commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator

@max-nextcloud heh we're using Postgres 😕 Why it's excluded?

The latter seems to have some discussions on creating the index by hand to avoid the DBAL limitations.

@Koc
Koc force-pushed the feature/optimize-pages-loading-v2 branch from 6937cb2 to 8806c19 Compare June 27, 2026 20:15
@Koc
Koc requested review from max-nextcloud and mejo- June 27, 2026 20:53
Koc added 3 commits June 29, 2026 23:12
@Koc
Perf: Optimize pages loading
7b33ea5 
Signed-off-by: Kostiantyn Miakshyn <molodchick@gmail.com>
@Koc
Perf: Optimize pages loading (code review fixes)
1f13cec 
Signed-off-by: Kostiantyn Miakshyn <molodchick@gmail.com>
@Koc
Perf: Optimize pages loading (code review fixes 2)
79f03d5 
Signed-off-by: Kostiantyn Miakshyn <molodchick@gmail.com>
@Koc
Koc force-pushed the feature/optimize-pages-loading-v2 branch from 795d7ac to 79f03d5 Compare June 29, 2026 21:16
@max-nextcloud

max-nextcloud commented Jun 30, 2026

Copy link
Copy Markdown
Collaborator

Thanks a lot for your patience and commitment @Koc
I'll take another look to be sure and hope I can approve the PR then.

max-nextcloud
max-nextcloud approved these changes Jun 30, 2026
View reviewed changes

@max-nextcloud max-nextcloud left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot! 💙

@max-nextcloud max-nextcloud mentioned this pull request Jun 30, 2026
Open
@max-nextcloud

max-nextcloud commented Jun 30, 2026

Copy link
Copy Markdown
Collaborator

Tracking behat failure in #2614

@max-nextcloud
max-nextcloud dismissed mejo-’s stale review June 30, 2026 12:15

Issues have been adressed. Follow up issue created.

@max-nextcloud
max-nextcloud merged commit 6906d5f into main Jun 30, 2026
84 of 93 checks passed
@max-nextcloud
max-nextcloud deleted the feature/optimize-pages-loading-v2 branch June 30, 2026 14:10
@github-project-automation github-project-automation Bot moved this from 👀 In review to ☑️ Done in 📝 Productivity team Jun 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@max-nextcloud max-nextcloud max-nextcloud approved these changes
@silverkszlo silverkszlo Awaiting requested review from silverkszlo silverkszlo is a code owner
@mejo- mejo- Awaiting requested review from mejo- mejo- is a code owner

Assignees

@Koc Koc

@max-nextcloud max-nextcloud

Labels

None yet

Projects

📝 Productivity team
Archived in project

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Optimize performance of the pages loading by stopping using filesystem always

3 participants

@Koc @max-nextcloud @mejo-